Skip to content

feat(native): Add iceberg partition transforms #26479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

feat(native): Add iceberg partition transforms #26479

wants to merge 2 commits into from

Conversation

@PingLiuPing
Copy link
Contributor

@PingLiuPing PingLiuPing commented Oct 30, 2025 •
edited
Loading

Description

Support Iceberg partition transforms: Identity, bucket, truncate, year, month, day, hour.

Motivation and Context

Impact

Test Plan

Contributor checklist

  • Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.
  • If adding new dependencies, verified they have an OpenSSF Scorecard score of 5.0 or higher (or obtained explicit TSC approval for lower scores).

Release Notes

Please follow release notes guidelines and fill in the release notes below.

Prestissimo (Native Execution) Changes
* Add iceberg partition transform support.

@prestodb-ci prestodb-ci added the from:IBM PR from IBM label Oct 30, 2025
@sourcery-ai
Copy link
Contributor

sourcery-ai bot commented Oct 30, 2025 •
edited
Loading

Reviewer's Guide

This PR implements support for Iceberg partition transforms by introducing new conversion routines that translate Presto’s IcebergPartitionSpec into Velox’s PartitionSpec and integrating these routines into the table-handle translation path.

Sequence diagram for Iceberg partition spec conversion during table handle translation

sequenceDiagram
    participant PrestoTableHandle
    participant IcebergPrestoToVeloxConnector
    participant VeloxPartitionSpec
    PrestoTableHandle->>IcebergPrestoToVeloxConnector: Provide IcebergPartitionSpec
    IcebergPrestoToVeloxConnector->>IcebergPrestoToVeloxConnector: toVeloxIcebergPartitionSpec(spec, typeParser)
    IcebergPrestoToVeloxConnector->>VeloxPartitionSpec: Return converted PartitionSpec
    IcebergPrestoToVeloxConnector->>PrestoTableHandle: Return VeloxPartitionSpec
Loading

Class diagram for IcebergPrestoToVeloxConnector partition transform support

classDiagram
    class IcebergPrestoToVeloxConnector {
        +toVeloxIcebergPartitionField(field, typeParser, schema) : IcebergPartitionSpec::Field
        +toVeloxIcebergPartitionSpec(spec, typeParser) : IcebergPartitionSpec*
    }
    class PrestoToVeloxConnector
    IcebergPrestoToVeloxConnector --|> PrestoToVeloxConnector
    class ProtocolIcebergIcebergPartitionField
    class ProtocolIcebergPrestoIcebergPartitionSpec {
        fields
        schema
        specId
    }
    class VeloxConnectorHiveIcebergIcebergPartitionSpec {
        specId
        fields
    }
    class VeloxConnectorHiveIcebergIcebergPartitionSpecField {
        name
        type
        transform
        parameter
    }
    IcebergPrestoToVeloxConnector --> VeloxConnectorHiveIcebergIcebergPartitionSpec
    IcebergPrestoToVeloxConnector --> VeloxConnectorHiveIcebergIcebergPartitionSpecField
    IcebergPrestoToVeloxConnector --> ProtocolIcebergIcebergPartitionField
    IcebergPrestoToVeloxConnector --> ProtocolIcebergPrestoIcebergPartitionSpec
Loading

File-Level Changes

Change Details Files
Add methods to convert Presto Iceberg partition fields and specs to Velox PartitionSpec
  • Declare toVeloxIcebergPartitionField and toVeloxIcebergPartitionSpec in the connector header
  • Implement toVeloxIcebergPartitionField to map field names, types, and transform parameters
  • Implement toVeloxIcebergPartitionSpec to aggregate fields into a PartitionSpec instance
 IcebergPrestoToVeloxConnector.cpp
IcebergPrestoToVeloxConnector.h
Integrate partition spec conversion into insert table handle translation
  • Invoke toVeloxIcebergPartitionSpec when building new and existing insert table handles
 IcebergPrestoToVeloxConnector.cpp
Update includes and namespaces to support new partition spec code
  • Add headers for Iceberg PartitionSpec and TableHandle
  • Include velox connectors for iceberg data sink
  • Introduce 'using namespace velox' for brevity
 IcebergPrestoToVeloxConnector.cpp
IcebergPrestoToVeloxConnector.h
Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@steveburnett
Copy link
Contributor

steveburnett commented Oct 30, 2025

Suggest revising the release note entry for formatting. Topic should probably be either
Prestissimo (Native Execution) Changes
or
Iceberg Connector Changes.

== RELEASE NOTES ==
Iceberg Connector Changes

* Add Iceberg partition transform support.

@PingLiuPing PingLiuPing force-pushed the lp_iceberg_partition_transform branch from d70f6c7 to 89ece31 Compare October 30, 2025 15:48
@PingLiuPing PingLiuPing changed the title feat: Add iceberg partition transforms feat(native): Add iceberg partition transforms Nov 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hantangwangd hantangwangd Awaiting requested review from hantangwangd hantangwangd will be requested when the pull request is marked ready for review hantangwangd is a code owner

@ZacBlanco ZacBlanco Awaiting requested review from ZacBlanco ZacBlanco will be requested when the pull request is marked ready for review ZacBlanco is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

from:IBM PR from IBM

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@PingLiuPing @steveburnett @prestodb-ci